Papers with visual representations
Seeing More, Saying More: Lightweight Language Experts are Dynamic Video Token Compressors (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for converting visual tokens into tokens are limited by their high volume . et al., 2023; Zheng e.t., 2023): a revolution in video understanding. |
| Approach: | They propose a language-aware dynamic token compression system that converts video clips into soft caption tokens as visual representations. |
| Outcome: | The proposed method reduces FLOPs by 49% while maintaining competitive performance. |
A Character-Centric Creative Story Generation via Imagination (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing narrative generation models lack diversity and character depth, but they are inadequate for human creativity. |
| Approach: | They propose a novel story generation framework called CCI that leverages images to create stories that are diverse and creative in their themes and richer in content. |
| Outcome: | The proposed framework significantly improves various aspects of the stories’ creativity. |
How agents see things: On visual representations in an emergent language game (D18-1)
Copied to clipboard
| Challenge: | Existing studies focus on the agents’ symbol usage, rather than on their representation of visual input. |
| Approach: | They propose to use visual representations of objects to create language-like communication systems by integrating them with the visual input of a game. |
| Outcome: | The proposed model and setup of Lazaridou et al. (2017) show that the representations of the agents' symbols do not capture the conceptual properties of the objects depicted in the input images. |
Mitigating Visual Knowledge Forgetting in MLLM Instruction-tuning via Modality-decoupled Gradient Descent (2025.findings-emnlp)
Copied to clipboard
Junda Wu, Yuxin Xiong, Xintong Li, Yu Xia, Ruoyu Wang, Yu Wang, Tong Yu, Sungchul Kim, Ryan A. Rossi, Lina Yao, Jingbo Shang, Julian McAuley
| Challenge: | Existing fine-tuning and continual learning methods compress visual representations and emphasize task alignment over visual retention. |
| Approach: | They propose a modality-decoupled gradient descent (MDGD) that regulates gradient updates to preserve effective rank of visual features and explicitly disentangles visual learning from task-specific alignment. |
| Outcome: | The proposed model reduces visual forgetting and improves visual retention . it disentangles visual learning from task-specific alignment and preserves effective rank . |
Localization vs. Semantics: Visual Representations in Unimodal and Multimodal Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing vision-and-language models perform better on multimodal tasks, but there is little understanding of how multimodal learning can help visual representations. |
| Approach: | They conduct a probing analysis of visual representations in existing vision-and-language models and vision-only models by probing on a broad range of tasks. |
| Outcome: | The proposed model improves vision-and-language models on label and attribute prediction tasks while vision-only models are stronger on dense prediction tasks. |
Efficient Table Retrieval and Understanding with Multimodal Large Language Models (2026.findings-eacl)
Copied to clipboard
| Challenge: | Tabular data is often captured in image form across a wide range of real-world scenarios. |
| Approach: | They propose a framework that enables MLLMs to answer queries over large tables. |
| Outcome: | The proposed framework outperforms existing methods by 7.0% in retrieval recall and 6.1% in answer accuracy on a newly constructed dataset with 48,504 unique tables. |
Synonym relations affect object detection learned on vision-language data (2024.findings-naacl)
Copied to clipboard
| Challenge: | a recent study shows that vision-language models that accept textual input are not robust to variations in how input is provided. |
| Approach: | They propose two approaches to improve vision-language object detectors' performance . they use back-translation and class embedding enrichment to improve their models . |
| Outcome: | The proposed approaches improve performance on synonyms from mAP@0.3=33.87% to 37.93%. |
UNIMO-2: End-to-End Unified Vision-Language Grounded Learning (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for vision-language pre-training can only learn from aligned image-caption data and rely heavily on expensive regional features. |
| Approach: | They propose an end-to-end unified-modal pre-training framework for joint learning . they propose to conduct grounded learning on both images and texts via a sharing grounded space . |
| Outcome: | The proposed model improves visual and visual semantic alignment on images and texts. |
Tell Me What’s Next: Textual Foresight for Generic UI Representations (2024.findings-acl)
Copied to clipboard
| Challenge: | Prior work has learned strong visual representations with local or global captioning losses, but fails to retain both granularities. |
| Approach: | They propose a pretraining objective for learning UI screen representations using captioning. |
| Outcome: | The proposed approach outperforms state-of-the-art on generation tasks with 28x fewer images. |
MIRe: Enhancing Multimodal Queries Representation via Fusion-Free Modality Interaction for Multimodal Retrieval (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods focus on textual queries that include visual information, but lack the ability to address multimodal queries that encompass both textual and visual information. |
| Approach: | They propose a retrieval framework that achieves modality interaction without fusing textual features during the alignment. |
| Outcome: | The proposed method achieves modality interaction without fusing textual features during the alignment. |
Enhancing Multimodal Entity Linking with Jaccard Distance-based Conditional Contrastive Learning and Contextual Visual Augmentation (2025.naacl-long)
Copied to clipboard
Cong-Duy T Nguyen, Xiaobao Wu, Thong Thanh Nguyen, Shuai Zhao, Khoi M. Le, Nguyen Viet Anh, Feng Yichao, Anh Tuan Luu
| Challenge: | Existing approaches to multimodal entity linking use contrastive learning to align input sentences and entities, but are limited by their random negative sampling. |
| Approach: | They propose a method to match negative samples with similar attributes using JD-CCL . they also propose 'contextual visual-aid controllable patch transform' experimental results demonstrate the strong effectiveness of their method . |
| Outcome: | The proposed method is able to match negative samples with similar attributes on a multimodal knowledge graph. |
VizoMem: A Visual-Textual Memory Framework for Efficient Long-Horizon Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing systems that use long-context modeling incur computational and memory overhead. |
| Approach: | They propose a visual memory framework that pre-rendered text into structured images and stored as visual notes for agentic systems. |
| Outcome: | The proposed system reduces token consumption while preserving effective long-term memory recall. |
Visual Storytelling with Question-Answer Plans (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models focus on enhancing the representation of image sequences, but the stories are repetitive, illogical, and lacking in detail. |
| Approach: | They propose a framework which integrates visual representations with pretrained language models and planning. |
| Outcome: | The proposed framework combines visual representations with pretrained language models and planning. |
Neural Machine Translation with Phrase-Level Universal Visual Representations (2022.acl-long)
Copied to clipboard
| Challenge: | Existing multimodal machine translation methods require paired input of source sentence and image, which makes them suffer from shortage of sentence-image pairs. |
| Approach: | They propose a phrase-level retrieval-based method to get visual information from existing sentence-image data sets. |
| Outcome: | The proposed method significantly outperforms strong baselines on multiple MMT datasets, especially when the textual context is limited. |
Encoding and Controlling Global Semantics for Long-form Video Question Answering (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to find answers for long videos fail to reason over the whole sequence of video, leading to sub-optimal performance. |
| Approach: | They propose a state space layer to integrate global semantics into video . they use a gating unit to enable controllability over the flow of global semantic into visual representations. |
| Outcome: | The proposed framework is able to integrate global semantics into visual representations. |
Probing Contextual Language Models for Common Ground with Visual Representations (2021.naacl-main)
Copied to clipboard
| Challenge: | Contextual language models have attracted great interest in probing what is encoded in their representations. |
| Approach: | They propose a probing model that evaluates how effective are text-only representations in distinguishing between matching and non-matching visual representations. |
| Outcome: | The proposed model outperforms text-only language models in instance retrieval, but underperform humans. |
BBA: Bi-Modal Behavioral Alignment for Reasoning with Large Vision-Language Models (2024.findings-acl)
Copied to clipboard
Xueliang Zhao, Xinting Huang, Tingchen Fu, Qintong Li, Shansan Gong, Lemao Liu, Wei Bi, Lingpeng Kong
| Challenge: | Multimodal reasoning is a key capability for large vision-language models . however, the vanilla Chain-of-Thought method fails to address critical steps in multi-step reasoning tasks. |
| Approach: | They propose a bi-modal Behavioral Alignment method to augment multimodal reasoning . they use domain-specific language to integrate multimodal information into a precise alternative form . |
| Outcome: | The proposed method significantly improves GPT-4V(ision) on geometry problem solving, chess positional advantage prediction and molecular property prediction. |
Shaping Visual Representations with Language for Few-Shot Classification (2020.acl-main)
Copied to clipboard
| Challenge: | Existing models use natural language descriptions to classify images, but no model uses it for new tasks. |
| Approach: | They propose a model that regularizes visual representations to predict language in a few-shot setting . they propose to use language to improve few- shot visual classification . |
| Outcome: | The proposed model outperforms baseline models in two challenging few-shot domains. |
On the Perception Bottleneck of VLMs for Chart Understanding (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a perception bottleneck in large vision-language models is critical for chart understanding . instruction tuning improves the extraction capability of LVLMs, but the vision encoder remains a bottleneck . |
| Approach: | They propose to decompose the perception bottleneck into two components . the vision encoder bottleneck is where visual representation fails to encapsulate the correct information . |
| Outcome: | The proposed approach significantly mitigates the vision encoder bottleneck and improves the ability of LVLMs to comprehend charts. |
Learning from Adjective-Noun Pairs: A Knowledge-enhanced Framework for Target-Oriented Multimodal Sentiment Classification (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods to determine sentiment polarity of opinion target are inconsistent and lack visual attention. |
| Approach: | They propose a framework which can exploit adjective-noun pairs extracted from images to improve visual attention and sentiment prediction capability of the TMSC task. |
| Outcome: | The proposed framework outperforms state-of-the-art on two public datasets. |
LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering (2025.acl-long)
Copied to clipboard
| Challenge: | Multimodal Large Language Models (MLLMs) enhance visual tasks by integrating visual representations into large language models. |
| Approach: | They propose a method to re-balance modalities by steering visual representations . they propose LLaVA Steering, a platform that enables rapid customization of MLLMs a component-based architecture . |
| Outcome: | The proposed model re-balances the modalities of visual representations in large language models . the model requires 500 times fewer trainable parameters than LoRA while maintaining comparable performance . |
LogogramNLP: Comparing Visual and Textual Representations of Ancient Logographic Writing Systems for NLP (2024.acl-long)
Copied to clipboard
| Challenge: | Existing pipelines for natural language processing only process symbolic representations of language, which are labor-intensive and noisy . a large portion of logographic data persists in a purely visual form due to the absence of transcription . this issue poses a bottleneck for researchers seeking to apply NLP to ancient logographic languages . |
| Approach: | They propose a benchmark for NLP analysis of ancient logographic languages using visual representations of writing. |
| Outcome: | The proposed pipeline outperforms existing pipelines for some tasks . the results could unlock large amounts of cultural heritage data of ancient logographic languages . |
LLMs Can Compensate for Deficiencies in Visual Representations (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a strong language backbone in vision-language models compensates for weak visual features by contextualizing or enriching them. |
| Approach: | They investigate whether strong language backbone compensates for weak visual features . they use CLIP-based vision encoders to perform controlled self-attention ablations . |
| Outcome: | The proposed model compensates for weak visual features by contextualizing or enriching them. |
Enabling Unsupervised Neural Machine Translation with Word-level Visual Representations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Unsupervised neural machine translation methods have been observed to make particular errors in comparison to supervised machine translation, such as confusing nouns that pertain to the same semantic category. |
| Approach: | They propose a method that incorporates images at the word level to augment lexical mappings. |
| Outcome: | Experiments on a multi-lingual dataset show that the proposed method generates more accurate translations with only monolingual data. |
UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for chart-based data analysis neglect explicit modeling of chart structures. |
| Approach: | They propose a pretrained model for chart comprehension and reasoning that encodes relevant text, data, and visual elements of charts and uses a chart-grounded text decoder for text generation. |
| Outcome: | The proposed model outperforms existing methods that lack explicit modeling of chart structures and lacks explicit modeling. |
nvAgent: Automated Data Visualization from Natural Language via Collaborative Agent Workflow (2025.acl-long)
Copied to clipboard
| Challenge: | *Natural Language to Visualization (NL2Vis) seeks to transform natural-language descriptions into visual representations of given tables. |
| Approach: | They propose a collaborative agent workflow for NL2Vis that incorporates three agents . the model is called **nvAgent** and comprises a processor agent for database processing and context filtering, a composer agent for planning visualization generation and a validator agent for code translation and output verification. |
| Outcome: | The proposed workflow surpasses state-of-the-art models on the VisEval benchmark. |
Unraveling the Truth: Do VLMs really Understand Charts? A Deep Dive into Consistency and Robustness (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Chart question answering (CQA) is a crucial area of Visual Language Understanding. |
| Approach: | They evaluate the robustness and consistency of current Visual Language Models on a dataset encompassing diverse question categories and chart formats. |
| Outcome: | The proposed models handle varying levels of chart and question complexity and are robust across different visual representations of the same underlying data. |
Fine-grained Medical Vision-Language Representation Learning for Radiology Report Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to learn medical vision-language representations by contrasting images with entire reports are not effective. |
| Approach: | They propose a phenotype-driven medical vision-language representation learning framework to bridge the gap between visual and textual modalities for improved text-oriented generation. |
| Outcome: | The proposed framework bridges the gap between visual and textual modalities for improved radiology report generation. |
CoViPAL: Layer-wise Contextualized Visual Token Pruning for Large Vision-Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to prune redundant vision tokens struggle in shallow layers due to the lack of contextual information. |
| Approach: | They propose a layer-wise contextualized visual token pruning method that uses a plug-and-play Pruning Module to prune redundant vision tokens. |
| Outcome: | The proposed method outperforms training-free pruning methods under equal token budgets and surpasses training based methods with comparable supervision. |
Unveiling the mystery of visual attributes of concrete and abstract concepts: Variability, nearest neighbors, and challenging categories (2024.emnlp-main)
Copied to clipboard
| Challenge: | a recent study examines the visual representation of concrete concepts using images from Bing and YFCC. |
| Approach: | They examine the variability in visual representations by using images of concrete and abstract concepts from Bing and YFCC. |
| Outcome: | The proposed model can distinguish between concrete and abstract concepts using basic visual features, the authors show . their model outperforms other models in the nearest neighbor analysis, but it is more complex and requires more visual features . |
Lost in Embeddings: Information Loss in Vision–Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Experiments reveal connectors substantially distort the local geometry of visual representations, with k-nearest neighbors diverging by 40–60% post-projection, correlating with degradation in retrieval performance. |
| Approach: | They propose two approaches to examine and quantify information loss by analyzing latent representation space. |
| Outcome: | The proposed model improves retrieval performance by analyzing changes in k-nearest neighbor relationships between image representations before and after projection. |
Mitigating Hallucination in Large Vision-Language Models through Aligning Attention Distribution to Information Flow (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Decode-Only models propagate information from left to right, but the model's attention still focuses on the visual representations, resulting in hallucinations. |
| Approach: | They propose to leverage the core information embedded in semantic representations to enhance the model's visual understanding by leveraging the attention distributions. |
| Outcome: | The proposed method reduces hallucinations by 80% by aligning the attention distribution with the actual information flow. |
Reducing Peak Memory Usage for Modern Multimodal Large Language Model Pipelines (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to reduce memory usage of multimodal large language models rely on storing large numbers of vision tokens in the key–value cache . however, such compression is typically only applied after all inputs are processed, resulting in high peak memory usage during the prefill stage. |
| Approach: | They propose a sequential input-compression mechanism that enforces a fixed memory budget by performing structure-aware key–value cache compression during the prefill stage. |
| Outcome: | The proposed approach reduces peak memory usage while maintaining generative performance with only minimal degradation, enabling more practical and memory-efficient multimodal inference. |
Visual-Linguistic Dependency Encoding for Image-Text Retrieval (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing approaches to image-text retrieval ignore semantic discrepancies caused by syntactic structure in natural language expressions and relationships among visual entities. |
| Approach: | They propose a visual-linguistic dependency encoder framework which explicitly models the dependency information among textual words and interaction patterns between image regions. |
| Outcome: | The proposed framework outperforms existing methods on a vision-linguistic compositional structure reasoning dataset. |
Unlocking Human-Like Visible Logic: How Logic Diagrams Boost Logic Reasoning in Large Language Models? (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have demonstrated their remarkable capabilities in natural language understanding and generation, but they struggle with formal logical reasoning. |
| Approach: | They propose to incorporate visual logic diagrams into LLMs’ reasoning workflows to enhance their performance on formal logic tasks. |
| Outcome: | The proposed model improves on syllogistic and conditional reasoning with programmatically generated Venn, Euler, and Linear diagrams. |
InsAT: Instance-aware Semantic Alignment and Transfer from Human–Object Keypoints for Zero-to-Few-shot Action Understanding (2026.acl-long)
Copied to clipboard
| Challenge: | Existing zero-shot (ZS) approaches emphasize human motion while underutilizing contextual information, particularly human–object interactions. |
| Approach: | They propose a framework for ZS recognition and zero-to-few-shot adaptation that leverages instance-level language descriptions. |
| Outcome: | The proposed framework outperforms keypoint-based ZS methods while remaining data-efficient and robust. |
Beyond Static Alignment: Adaptive Arbitration for Semantic Incongruence in Semi-Supervised Multimodal Sentiment Analysis (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for semantic incongruence in sentiment analysis are limited by label-limited settings. |
| Approach: | They propose a framework for semi-supervised multimodal sentiment analysis that emphasizes stable cross-modal representations and reliable supervision. |
| Outcome: | The proposed framework outperforms state-of-the-art methods under label-limited settings. |
Beyond Polarity: Continuous Affect-Enhanced Multimodal Aspect-Based Sentiment Classification (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for multimodal aspect-based sentiment classification exploit discrete polarity patterns and generic visual embeddings. |
| Approach: | They propose a Valence–Arousal–Dominance(VAD)-Enhanced MABSC framework that integrates VAD signals into multimodal sentiment reasoning and learns emotion-sensitive image representations. |
| Outcome: | The proposed framework brings VAD signals into multimodal sentiment reasoning and learns emotion-sensitive image representations. |